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Abstract. We compute the number of all rhombus tilings of a hexagon with sides a, 6 + 
1, c, a + 1, 6, c + 1, of which the central triangle is removed, provided a, b, c have the same parity. 
The result is B^], [fl, rfl)i?(r^l, [fj , [f 1 )B( [f 1 , T^l, Lf J)S(Lf J, [fl, T^l). where 
B{a,/3,'y) is the number of plane partitions inside the a x /3 x 7 box. The proof uses nonin- 
tersecting lattice paths and a new identity for Schur functions, which is proved by means of 
the minor summation formula of Ishikawa and Wakayama. A symmetric generalization of this 
identity is stated as a conjecture. 



1. Introduction. In recent years, the enumeration of rhombus tilings of various regions has 
attracted a lot of interest and was intensively studied, mainly because of the observation (see 
[8]) that the problem of enumerating all rhombus tilings of a hexagon with sides a, 6, c, a, 6, c 
(see Figure 1; throughout the paper by a rhombus we always mean a rhombus with side 
lengths 1 and angles of 60° and 120°) is another way of stating the problem of counting 
all plane partitions inside an a x 6 x c box. The latter problem was solved long ago by 
MacMahon [10, Sec. 429, q ^ 1, proof in Sec. 494]. Therefore: 

The number of all rhombus tilings of a hexagon with sides a, 6, c, a, b, c equals 



i=i j=i k=i 
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(The form of the expression is due to Macdonald.) 

In his preprint [12], Propp proposes several variations of this enumeration problem, one 
of which (Problem 2) asks for the enumeration of all rhombus tilings of a hexagon with sides 
n, n + 1, n, n + 1, n, n + 1, of which the central triangle is removed (a "punctured hexagon" ). 
At this point, this may seem to be a somewhat artificial problem. But sure enough, soon 
after, an even more general problem, namely the problem of enumerating all rhombus tilings 
of a hexagon with sides a, 6 + 1, c, a + 1, 6, c + 1, of which the central triangle is removed (see 
Figure 2), occured in work of Kuperberg related to the Penrose impossible triangle [private 
communication] . 

The purpose of this paper is to solve this enumeration problem. (We want to mention 
that Ciucu [2] has an independent solution of the b — c case of the problem, which builds 
upon his matchings factorization theorem [1].) Our result is the following: 

Theorem 1. Let a, 6, c be positive integers, all of the same parity. Then the number of all 
rhombus tilings of a hexagon with sides a,b+l,c,a + l,b,c+l, of which the central triangle 
is removed, equals 
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A rhombus tiling of a "punctured" hexagon 
with sides a,b + l,c,a + l,b,c + 1, where a = 3, b — 5, c — 5 

Figure 2 



where B{a, P, 7) is the number of all plane partitions inside the a x /5 x 7 box, which is given 
by (1.1). 

We prove this Theorem by first converting the tihng problem into an enumeration problem 
for nonintcrsccting lattice paths (see Section 2), deriving a certain summation for the desired 
number (see Section 3, Proposition 2), and then evaluating the sum by proving (in Section 4) 
an actually much more general identity for Schur functions (Theorem 3). The summation 
that we are interested in then follows immediately by setting all variables equal to 1. In order 
to prove this Schur function identity, which we do in Section 5, we make essential use of the 
minor summation formula of Ishikawa and Wakayama [5] (see Theorem 7). In fact, it appears 
that a symmetric generalization of this Schur function identity (Conjecture 5 in Section 4) 
holds. However, so far we were not able to establish this identity. Also in Section 4, we 
add a further enumeration result, Theorem 4, on rhombus tilings of a "punctured" hexagon, 
which follows from a different specialization of Theorem 3. 

In conclusion of the Introduction, we wish to point out a few interesting features of our 
result in Theorem 1. 

(1) First of all, it is another application of the powerful minor summation formula of 
Ishikawa and Wakayama. For other applications see [6, 7, 11]. 



4 



S. OKADA AND C. KRATTENTHALER 



(2) A very striking fact is that this problem, in the formulation of nonintersecting lattice 
paths, is the first instance of existence of a closed form enumeration, despite the fact that 
the starting points of the paths are not in the "right" order. That is, the "compatability" 
condition for the starting and end points in the main theorem on nonintersecting lattice 
paths [4, Cor. 2; 14, Theorem 1.2] is violated (compare Figure 3.c) and the main theorem 
does therefore not apply. (The "compatability" condition is the requirement that whenever 
we consider starting points Ai, Aj, with i < j, and end points E^, Ei, with k < I, then any 
path from Ai to Ei has to touch any path from Aj to Ek-) This is the reason that we had 
to resort to something else, which turned out to be the minor summation formula. 

(3) The result (1.2) is in a very appealing combinatorial form. It is natural to ask for a 
bijective proof of the formula, i.e., for setting up a one-to-one correspondence between the 
rhombus tilings in question and a quadruple of plane partitions as described by the product 
in (1.2). However, this seems to be a very difficult problem. In particular, how does one 
split a rhombus tiling of a hexagon into four objects? 

(4) In the course of our investigations, we discovered the symmetric generalization. Con- 
jecture 5, of Theorem 3 which we mentioned before. Although the minor summation formula 
still applies, we were not able to carry out the subsequent steps, i.e., to find the appropriate 
generalizations of Lemmas 9 and 10. 

2. Prom rhombus tilings to nonintersecting lattice paths. In this section we trans- 
late our problem of enumerating rhombus tilings of a "punctured" hexagon into the language 
of nonintersecting lattice paths. Throughout this section we assume that a, b, c are positive 
integers, all of the same parity. 

We use a slight (but obvious) modification of the standard translation from rhombus 
tilings of a hexagon to nonintersecting lattice paths. Given a rhombus tiling of a hexagon 
with sides a, 6 + 1, c, a + 1, 6, c + 1, of which the central triangle is removed (see Figure 2), 
we mark the centres of the a edges along the side of length a, we mark the centres of the 
a+1 edges along the side of length a + 1 , and we mark the centre of the edge of the removed 
triangle that is parallel to the sides of respective lengths a and a + 1 (see Figure 3. a; the 
marked points are indicated by circles). Starting from each of the marked points on the 
side of length a, we form a lattice path by connecting the marked point with the centre 
of the edge opposite to it, the latter centre with the centre of the edge opposite to it, etc. 
(see Figure 3.a,b; the paths are indicated by broken lines). For convenience, we deform the 
picture so that the slanted edges of the paths become horizontal (sec Figure 3.c). Thus we 
obtain a family of lattice paths in the integer lattice Z^, consisting of horizontal unit steps 
in the positive direction and vertical unit steps in the negative direction, with the property 
that no two of the lattice paths have a point in common. In the sequel, whenever we use 
the term "lattice path" we mean a lattice path consisting of horizontal unit steps in the 
positive direction and vertical unit steps in the negative direction. As is usually done, we 
call a family of lattice paths with the property that no two of the lattice paths have a point 
in common nonintersecting . 

It is easy to see that this correspondence sets up a bijection between rhombus tilings of 
a hexagon with sides a,b+l,c, a+l,b,c+l, of which the central triangle is removed, and 
families (Pi, P2, • • • , Pa+i) of nonintersecting lattice paths, where for i = 1, 2 . . . , a the path 
Pi runs from Ai — c+i) to one of the points Ej — (b+j — l,j — l), j — 1,2, ... , a+1, and 
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c. After deformation: A family of nonintersecting lattice paths 

Figure 3 
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where Po_|_i runs from = ((a + 6)/2, (a + c)/2) to one of the points E'j = — — 1), 
j = 1,2, . . . ,a + 1. It is the latter enumeration problem that we are going to solve in the 
subsequent sections. 

As mentioned in the Introduction, this enumeration problem for nonintersecting lattice 
paths is unusual, as the starting points are not lined up in the "right" way (in this case 
this would mean from bottom-left to top-right), the (a -|- l)-st starting point being located 
in the middle of the region. So, it is not fixed which starting point is connected to which 
end point. In our example of Figure 3.c, Ai is connected with Ei, A2 is connected with 
E2, A3 is connected with E^, and the "exceptional" starting point is connected with Es. 
But there are other possibilities. In general, for any /c, 1 < /c < a -|- 1, it is possible that 
Ai is connected with Ei, . . . , A^ is connected with Ej., A^+i is connected with £'^+2, ■ ■ ■ , 
Aa is connected with Ea+i, and the "exceptional" starting point Aa+i is connected with 
Ek^i. To obtain the solution of our enumeration problem, we have to find the number of 
all nonintersecting lattice paths in each case and then form the sum over all k. 

3. From nonintersecting lattice paths to Schur functions. The aim of this section 
is to describe, starting from the interpretation of our problem in terms of nonintersecting 

lattice paths that was derived in the previous section, how to derive an expression for the 
number of rhombus tilings that we are interested in. This expression, displayed in (3.4), 
features (specialized) Schur functions. 

Usually, when enumerating nonintersecting lattice paths, one obtains a determinant by 
means of the main theorem on nonintersecting lattice paths [4, Cor. 2; 14, Theorem 1.2]. 
However, as mentioned in the Introduction, the arrangement of the starting points in our 
problem, as described in the previous section, does not allow a direct application of this 
theorem. 



P3 




Figure 4 

In order to access the problem, as formulated at the end of the previous section, we set it 
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up as follows. Each of the lattice paths Pi, i = 1,2, ... ,a, has to pass through a lattice point 
on the diagonal line x — y = {b — c)/2, the central point Aa+i = ((a+6)/2, (a+c)/2) (which is 
the starting point for path Pa+i) excluded (see Figure 4; the points on the diagonal where the 
paths may pass through are indicated by bold dots). So, as in the previous section, let k be 
an integer between 1 and a + 1. For any fixed choice of points Mi, . . . , Mk, Mk+2, ■ ■ ■ , Ma+i 
on the diagonal x — y = (6 — c)/2, such that Mi is to the left of M2, . . . , Mk is to the left 
of the central point ^0+1 5 ^0+1 is to the left of Mk+2, • • • , Ma is to the left of Ma+i, the 
number of families (Pi, P2, ■ ■ ■ , Pa+i) of nonintersecting lattice paths, where for i = 1, 2 . . . , 
the path Pi runs from Ai through Mi to Ei, where for i = k + 1, k + 2, . . . ,a the path Pi 
runs from Ai through Mj_|_i to Ei+i, and where Pa+i runs from Aa+i to Ek+i, is easily 
computed using the main theorem on nonintersecting lattice paths. For convenience, let 
Mk+i = Aa+i. Then this number equals 

det {\V{Ai ^ M,.+^(,>fc+i))|) • det (|P(M, ^ E^)\) , (3.1) 

where \V{A B) \ denotes the number of all lattice paths from A to B, and where x is the 
usual truth function, x{-^) = 1 if ^ is true and x(.A) = otherwise. 

It is now no difficulty to observe that the expression (3.1) can be rewritten using (special- 
ized) Schur functions. For information on Schur functions and related definitions we refer 
the reader to Chapter I of Macdonald's classical book [9]. There are many ways to express 
a Schur function. The one we need here is the Nagelsbach-Kostka formula (the "dual" 
Jacobi-Trudi identity; see [9, Ch. I, (3.5)]). Let A be a partition with largest part at most 
m. Then the Schur function sx{xi, X2, ■ ■ ■ , x^) is given by 

S\{xi,X2, . . . ,Xn) = det ie(tX)^-i+j{Xl,X2,. . .,Xn)), 
l<i,j<'m 

where es{xi, . . . , Xn) '■= J2i<ii<,...<ci^<n^h '"'^is is the elementary symmetric function of 
order s in xi, . . . ,Xn, and where *A denotes the partition conjugate to A. We write briefly 
Xn for the set of variables {xi, X2, ■ ■ ■ , Xn}. In particular, the symbol sx{Xn) will be short 

for Sx{xi,X2,...,Xn). 

Now suppose that Mg = ((a + b)/2 + i^, {a + c)/2 + igj, £ = 1,2, . . . ,a + 1. In particular, 
since Mk+i — Aa+i — ((a + b)/2, (a + c)/2), we have ik+i = 0. The ordering of the points 
Mi from left to right as i increases implies —{a + b)/2<ii < ■ ■ ■ < ik < ik+i = < ik+2 < 
■ ■ ■ < ia+i < {a + b)/2. Then it is straight-forward to see that, using the above notation, 
the expression (3.1) equals 

Sa(X(6+c+2)/2) • S^(^(6+c)/2) , (3.2) 

a;i=a;2 = ---=a;(b+c+2)/2 = l 

where A and fi are the partitions whose conjugates *A and are given by 



f b — a . u e u _ -1 

Ah— — ^ 1- la+l-h+xia+l-h>k+l) + " lOr ft — 1, . . . , a, 

— TT- - ih + h - 1 for /i = 1, . . . , a -I- 1. 



(3.3) 
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(In view of the geometric interpretation of the Nagelsbach-Kostka formula in terms of non- 
intersecting lattice paths, see [3, Sec. 4, Fig. 8; 13, Sec. 4.5], this could also be read off 
directly from the lattice path picture of our setup, as exemplified by Figure 4.) 

According to the preceding considerations, what we would like to do in order to find 
the total number of all rhombus tilings of a hexagon with sides a, 6 + 1, c, a + 1, 6, c + 1, of 
which the central triangle is removed, is to sum the products (3.2) of Schur functions over 
all possible choices of A and That is, we want to find the sum 



c+2)/2) • S/.(^(6+c)/2)) 



xi=a;2 = ---=a;((,+c+2)/2 



=1 



where (A, fi) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some 

k and zi, ^2, • ■ • , ia+i as above. 

Summarizing, we have shown the following. 

Proposition 2. Let a, 6, c be positive integers, all of the same parity. Then the number 
of all rhombus tilings of a hexagon with sides a,b+l,c,a + l,b,c + l, of which the central 
triangle is removed, equals 

J] (sa(^(6+c+2)/2) •Sm(^(6+c)/2)) , (3-4) 

. a;i=a;2 = -"=a;((,_,_c+2)/2=l 

where (A, /i) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some 
k with 1 < k < a + 1, and for some ii,i2, ■ ■ ■ , ia+i with —{a + b)/2<ii < • • • < < i^+i = 
< ik+2 < ■ ■ ■ < ia+i < (o + b)/2. 

4. The main theorem and its implications. As a matter of fact, even the unspecial- 
ized sum that appears in (3.4) can be evaluated in closed form. This is the subject of the 
subsequent Theorem 3, which is the main theorem of our paper. Theorem 1 of the Introduc- 
tion then follows immediately. Before we proceed to the proof of Theorem 3 in Section 5, 
in this section we formulate a conjectured generalization of Theorem 3 as Conjecture 5, 
and we complement the enumeration result in Theorem 1 by a further result (Theorem 4) 
on the enumeration of rhombus tilings of a "punctured" hexagon, which follows also from 
Theorem 3. 

Theorem 3. Let a, b, n be positive integers, where a and b are of the same parity. Then 

(A,m) 

= ^(r(o+l)/2lL''/2J)(^n) •S(ra/21 l'>/2^JXn) • S.r ,3] Fb/si ) (^n+l) " S/i ^ /aj r(b+i)/21 ) (Xn+i), 

(4.1) 

where (A,//) ranges over all possible pairs of partitions such thai (3.3) is satisfied, for some 
k with 1 < k < a + 1, and for some ii,i2, ■ ■ ■ , ia+i with —{a + b)/2<ii < • • • < if. < ik+i = 
< ik+2 < ■ ■ ■ < ia+i < (a + b)/2. 

Theorem 1 follows immediately from Proposition 2 and the n — [b + c)/2 special case of 
Theorem 3 by using the well-known fact (see [9, Ch. I, Sec. 5, Ex. 13. (b), g — > 1]) that the 
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evaluation of a Schur function S(^b)(X„) of rectangular shape at 
counts the number of all plane partitions inside the A x B x {n — B) box. 

A different specialization of Theorem 3 leads to an enumeration result for rhombus tilings 
of a "punctured" hexagon with sides a, & + 1, c, a + 1, 6, c + 1 where not all of a, b, c are of 
the same parity. Namely, on setting xi = X2 = ■ ■ ■ — Xn = 1, Xn+i — 0, n — {b + c)/2 in 
Theorem 3, and on finally replacing c by c+ 1, we obtain the following result by performing 
the analogous translations that lead from rhombus tilings to nonintersecting lattice paths 
and finally to specialized Schur functions. 



c+ 1< 




A "punctured" hexagon with sides a,b + 1, c, a + l,b, c + 1, 
c being of parity different from the parity of a and b 

Figure 5 



Theorem 4. Let a, b, c be positive integers, a and b of the same parity, c of different parity. 
Prom the hexagon with sides a,b-\- l,c, a + 1,6, c + 1 the triangle is removed, whose side 
parallel to the sides of lengths a and a + 1 lies on the line that is equidistant to these sides, 
whose side parallel to the sides of lengths b and 6 + 1 lies on the line that is by two "units" 
closer to the side of length 6 + 1 than to the side of length b, and whose side parallel to the 
sides of lengths c and c + 1 lies on the line that is by one "unit" closer to the side of length 
c + 1 than to the side of length c, see Figure 5. Then the number of all rhombus tilings of 
this "punctured" hexagon with sides a, 6+l,c, a+1,6, c+1 equals 



B 



a + 2 



c + 2 



B 



a + 1 



6 + 1 



c + 1 



B 



6 + 2 



(4.2) 

where B{a, /3, 7) is the number of all plane partitions inside the a x /3 x 7 box, which is given 
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by (1.1). 

Even if Theorem 3 suffices to prove Theorem 1 (and Theorem 4) , it is certainly of interest 
that apparently there holds a symmetric generalization of Theorem 3. We have overwhelming 
evidence through computer computations that actually the following is true. 

Conjecture 5. Let a, b, n be positive integers, where a and b are of the same parity. Then 

-S/^iXn) = S([(a+l)/2]L''/2J)(-^n) " S(|-a/2] r''/21 ) (-'^n U {oJ^+l }) 

(A,m) 

^ *([a/2]r«>/2i)(^n U {a;n+2}) • S( L^/aj r(b+i)/2i ) (^n+2) , (4.3) 

where (A,//) ranges over all possible pairs of partitions such that (3.3) is satisfied, for some 
k with 1 < A; < a + 1, and for some ii, i2, ■ ■ ■ jia+i with — (a + b) /2 < ii < • • • < ik < ik+i = 
< ik+2 < < ia+i < (a + b)/2. 

We remark that the specialization n = (b + c)/2, xi = X2 = ■ ■ ■ = ^C(6+c+4)/2 = 1 
of Conjecture 5 leads, up to parameter permutation, again to Theorem 4, thus providing 
further evidence for the truth of the Conjecture. 

5. The minor summation formula and proof of the main theorem. This final 
section is devoted to a proof of Theorem 3. It makes essential use of the minor summation 
formula of Ishikawa and Wakayama. An outline of the proof is as follows. First, the minor 
summation formula is used to convert the sum on the left-hand side of (4.1) into a Pfaffian. 
This Pfaffian can be easily reduced to a determinant. In Lemma 9 it is seen that this 
determinant factors into a product of two Pfaffians. Finally, Lemma 10 shows that each of 
these Pfaffians is, basically, a product of two Schur functions, so that the final form of the 
result, as given by the right-hand side of (4.1), follows from a simple computation, by which 
we conclude this section. 

To begin with, let us recall the minor summation formula due to Ishikawa and Wakayama 
[5, Theorem 2]. 

Theorem 7. Let n, p, q be integers such that n + q is even and < n — q < p. Let G be any 
n X p matrix, H be any n x q matrix, and A = (aij)i<i,j<p be any skew- symmetric matrix. 
Then we have 

J]Pf (A|^) det {Gk i H) = ^-ifi^-^V^Vi (^J^^ ^ ) , (5.1) 

where K runs over all (n — q)-element subsets of [l,p], A^ is the skew- symmetric matrix 
obtained by picking the rows and columns indexed by K and Gk is the sub-matrix of G 
consisting of the columns corresponding to K. 

In order to apply this formula, we have to first describe the pairs (A, //) of partitions over 
which the sum in (4.1) is taken directly. This is done in Lemma 8 below. The reader should 
be reminded that the description that the formulation of Theorem 3 gives is in terms of the 
conjugates of A and ^ (compare (3.3)), which is not suitable for application of the minor 
summation formula (5.1). 



THE NUMBER OF RHOMBUS TILINGS OF A "PUNCTURED" HEXAGON 



11 



For convenience, let TZ{a, b) be the set of all pairs (A,/x) of partitions satisfying (3.3), for 
some k with 1 < A; < a + 1, and for some h, 12, ... , ia+i with —(a + b)/2 < ii < ■ • ■ < ik < 
ik+i = < ik+2 < ■ ■ ■ < ia+i < (a + b)/2. 

First note that 1{X) < 6 + 1 and < b for (A,//) e n{a, b). 

Lemma 8. If we define subsets J = {ji < ■ ■ ■ < jb+i} and J' = {j[ < ■ ■ ■ < j'^^} by the 

relations 

jh = Xb+2-h + h-l forh=l,...,b+l, 
j'f^ = fib+i-h + h - 1 forh=l,...,b, 

then we have 

(1) {a + b)/2 e J. 

(2) J' = {a + b^j:jeJ,j^{a + b)/2}. 
Proof. (1) Since iA;+2 > 1 and < —1, we have 

b — a . , _^ a + b , 
Aa-fc = 1- ik+2 + a- k> — fc + 1, 



2 ■ ' '~ - 2 " ' 

b — a a + b 
\a-k+i = — Pi \-ik + a-k + l< — A; 



Hence we obtain A(„_|_;,)/2-fc+i — a — k, which means j(b-a)/2+k+i = {a + b)/2. 
(2) It is enough to show that 

Xb+2-h + fJ'h = a + ioT h = l,...,^-^ + k, 

h — Q- 

Xb+2-h + l^h-i = a foTh=— h A; + 2, . . . ,6+ 1. 

By (3.3), the parts of A and can be expressed in terms of the ij's: 

Xb+2-h = # |i : ia+l-j+xia+l-j>k+l) > H 2 - /j, - j| , 

f b — a 



If h < {b — a)/2 + k, then ik < {a + b)/2 + 2 — h — (a + l — k) and so we have Xb+2-h < a — k. 
This implies 

Xb+2-h = # I j : ij > -h + j^ 

and Xb+2-h + A*h = o + 1. Similarly, since iJ,h-i < k for h > {b — a) /2 + k + 2, we have 

Xb+2-h = # |i : ij+x{j>k+i) > ^-y^ -h + j + l^, 
r b-a ^ } 
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and Xb+2-h + (J'h-i = a. □ 

Now we describe our choices of matrices G, H and A for our application of the minor 
summation formula in Theorem 7. For a subset I = {ii < ■ ■ ■ < ip} of nonnegative integers, 
let Mi{Xn) be the n x p matrix with {k,l) entry x]^. We define three subsets P, Q, R as 
follows: 

n — 6, n — 6 + l,...,n — 6 + — 1, n — 6 + + 1, . . . , n — 6 + (a + 6) 
g= <'0, l,...,n-6-l,n-6+^^^ 



i? = {0,l,...,n-6 - 1}. 
Then let the matrices G and be given by 



/Mp(X„+i) \ 

1^ Mp{Xn)) 

/Mq(X,+i) 

V MR{Xn) 



Furthermore, we define 

r=|o,l,...,^-l,4:^ + l,...,a + 6 

Let A be the skew-symmetric matrix whose rows and columns are indexed by the set 

rur = ru{^: /c e F} 

and whose nonzero entries are given by 

1 if 1 < /c < (a + 6)/2 - 1 
-1 if (a + 6)/2 + 1 < /c < a + 6. 



k,a+b—k 



We apply Theorem 7 to these matrices G, H and A. 

For a 26-element subset K of F U F, the sub-Pfaffian Pf(A|^) is easily computed. This 
sub-Pfaffian vanishes unless the number of the unbarred elements in K is equal to that of 
the barred elements in K. UK — {ji, . . . , jf,, j(, . . . , j^}, then we have 



Pf(Af) 



(_l)#{/.:i.>(a+6)/2+l} + ^ ^ + ^ 

otherwise. 



Now recall the bideterminantal expression for Schur functions (see [9, Ch. I, (3.1)]), 

det {xf^''-') det {xf^''-') 

l<i,j<n 
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Then for a subset K as above, we have 

where A (resp. /i) is the partitfon corresponding to the subset J = {ji, . . . ,jb, (a + b)/2} 
(resp. J' = {a + b - ji, . . . , a + b - jb}) and A{Xn) = ni<i<j<n(^i ~ ^0- Therefore, if we 
apply the minor summation formula in Theorem 7, we obtain 



Sx{Xn+l)s^{Xn) = 



(-1) 



hn 



(A,M)e7^(a,6) 



A(X„+i)A(XO 



Pf{A^)det{Gk\H) 



Kcrur,\K\=2b 



(^_l)b"+(2n-26+l)(2n-26)/2 f G A H 



A(Xn+i)A(X^) 







(-1) 



bn+n—b 



If we write A 



GAX} H 
-^H 



B 
B 







A(X^+i)A(X, 
then we have 



Pf 



GA *G H 




(5.3) 



-Mp{Xn)B'Mp{Xn+i) 









By permuting rows and columns by the permutation 









1 ••• n + 1 n + 2 
1 ••• n+1 2n-6 + 2 



3n-6 + 2 3n-6 + 3 
4n - 26 + 2 n + 2 





MR{Xn) 





4n - 26 + 2 
2n - 6 + 1 



we have 
Pf 



GA *G H 

-^H 

(^_l)(2'^-&+l)(n-6) 

f 



-Mp(X„)S*Mp(X„+i) MniXr 
\ -*Mq(X„+i) 



X Pf 



Mp(X,+i) S *Mp(X,) Mq(X„+i) 

-'MniXr^) 







— ( _-[\bn-n+{2n-b+l){2n-b)/2 ^ . 



N{Xn,X, 

^_^^bn+b{b-l)/2+in-b) I -tMniX, 



Mp^Xn) M{n-b+{a+b)/2}{Xn) 





.N{Xn+l,Xn) Mp{Xn+l) 



X 



n-b+{a+b)/2 
n+1 



(5.4) 
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where 

iV(X^, Yn) = Mp{X^) B *Mp{Yn). 
By direct computation, the (i, j)-entry of N{Xjn, in) is equal to 



Here, the last determinant in (5.4) can be decomposed into the product of two Pfaffians 
by using the following lemma. 

Lemma 9. Let A be annxn skew-symmetric matrix, 6 = . . . , bn) and c = *(ci, . . . , c^) 
be column vectors, and d a scalar. Then the determinant of the (n + 1) x (n + 1) matrix 



A b 
-*c d 



decomposes into the product of two Pfaffians as follows: 
(1) If n is even, then 



Abe 

detyl= -Pf(y4)Pf I -*6 -d\. (5.5) 

-*c d 



(2) If n is odd, then 



det^ = Pt|ij ^)pf(_t D- (8.6) 



Proof. Expanding along the last column and the bottom row, we see that 

n ^ 

detl= ■ (-l)"+^-2(-c,) • det (aI^ + ddei{A) 

n ^ 

^ {-iy+^b,Cj det i^Al^ + ddet(A), 



where M denotes the matrix obtained from A by deleting the z-th row and the j-th. column. 

First suppose that n is even. Then, since Ai, (resp. A) is a skew-symmetric matrix of odd 
(resp. even) degree, det — (resp. det(A) = Pf(A)^). By induction hypothesis, we 
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see that, if i < j, then 



det (^Apj = (-l)("-^-i)+(--J)det 



Cljfi Clji I 



( 



= (_l).+.-l.(_l)pf pf 



aij \ 



an 



(liYi Ojj 



= (-1)*+^' • Pf (^Al'l) Pf(A) 

= -pf(4|) pf(^). 



Similarly, if z > then we have 



det {m^ =Pf Pf(A). 



Hence we have 



det A 



= pf(A)( (-ir+^+'^>^c,pf(4;i)+ 5^ (-ir+^6,c,pf(4|)+dPf(A) 

\ l<i<j<n l<i<j<n 

By comparing this expression with the expansion of the second Pfaffian in (5.5) along the 
last two columns. 



Abe 
Pf I -d 

-*c d 



= E (-ir^c,-(-iri6,pf(4|)+ e (-ir^c,-(-ir%pf(4;|) 



l<j<ji<n 
+ M)Pf(A). 



l<j<i<n 



we obtain the desired formula. 

Next suppose that n is odd. Then det A = Q. By induction hypothesis, we see that, if 
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i <j, 



det (^Apj = (-l)("-»-i)+("-i)det 

v 



\aji 



(_l)i+i-ipf 



an ^ ( 

Pf 



din / 



V. 



u / \ajx 

i+j — 1 / i\n — i — 1 



ajn / 



Pf(4)pf(4). 



Similarly, for i > j, we have 



det (aTJ = Pf (4) Pf 



Comparison of this with the expansion of the PfafSans in (5.6) along the last columns 
completes the proof. □ 

Now we apply this lemma. If b is even, then we have 



n—b+(a 



det I -'Mr{X^) 

^N{Xn+l,Xn) MR{Xn+l) 

_ (N{Xn,X^) Mr{X, 
\ -'MniXr,) 

/ N{Xn,Xn) 

-'MniXr,) 

*M^n-b+{a+b)/2} i^n) 
\ N{Xn+l,Xn) 

_ fN{X^,X^) Mr{X,, 

\ -'MR{Xn) 



X Pf 



X 





-b+{a+b)/2 



n+1 



MR(Xn) M^n_i,+ ^a+b)/2}{Xn) N{Xn,Xn+l) \ 






n-b+{a+b)/2 



-X 



*MR{Xn+l) 
n-b+{a+b)/2 



n+1 



MR{Xn+l) a;„+i 







X Pf [ N{Xn+l, Xn+l) MQ{Xn+l)\ 



I 



(5.7) 
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If b is odd, then we have 



det I -'Mr{X^) 

n— b+{a+b)/2}{Xn) 

-'MniXn) 

-*M^n-b+{a+b)/2}{^n) 
/ N{Xr,,Xn) Mn{Xn) iV(X„, 
xPf -'MniXn) -'MRiXn+l) 

\N{Xr,+i,Xr,) MniXr^+i) x::^-\+^''+'^/^ 

(N{X^,X^) Mq{X^ 

\-'MQ{Xr,) 



Pf 



X 



/ -,\n-b -pf ( ^i^n+l, Xn+l) Mji{Xn+l)\ 



The four Pfaffians in (5.7) and (5.8) are evaluated by the following Lemma. 
Lemma 10. (1) If a and b is even, then 

fN{Xr,,X^) M^(X,)\ 
1^ -'Mn{X^) ; 

^_l^bin-b)+{n-b)(n-b-l)/2 
" A{Xn) 

X det ^M[o,n-6/2-l](^n) '■ -^[n+a/2-6/2,n+a/2-l] (^n)^ 



X 



det ^M[o,n-6/2-l](-^n) : M[n+a/2-b/2+l,n+a/2]{Xn)^ , 



Pf 



f N{X^+^,Xr,+l) MQ{Xn+l) 

\ -*Mq(X,+i) 

(_l)fe(^-6)+&/2+(ri-6+l)(n-6)/2 



A(X,+i) 

X det ^M[o,n_b/2](-'fn+l) : Myn+a/2-b/2+l,n+a/2\{Xn+l)^ 
X det ^M[o,n-b/2-l](-'^n+l) '■ -/W"[n+a/2-&/2,n+a/2] (-^^n+l)^ 
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(2) If a and b is odd, then 
Pf 



\-'Mq{X^) 

(_l)(b-l)("-^') + (b-l)/2+(n-6+l)(n-&)/2 



X det ^M[o,n-(6-l)/2-l](-^n) '■ -^[n+(o+l)/2-(6-l)/2,n+(a+l)/2-l] (-^n)^ 
X det ^M[o,n-(6+l)/2-l](-^n) '■ -^[n+(a+l)/2-(6+l)/2,n+(a+l)/2-l] (-'^n)^ 



Pf 



(N{Xn+uXn+i) MR{Xn+i) 
\ -^MniXn+i) 

^_l-^{b+l){n-b) + {n-b){n-b-l)/2 



X det ^M[o^n-{b+l)/2]{Xn+l) '■ M[n+{a-l)/2-{b+l)/2+l,n+{a-l)/2]{Xn+l)^ 
X det ^M[o,n-(6+l)/2](-'fn+l) '■ -^[n+(o+l)/2-(6+l)/2+l,n+(a+l)/2l (-^^n+l)^ • 

Proof. We use Theorem 7 and the following decomposition of the product of two Schur 
functions of rectangular shape [11, Theorem 2.4]: Let m < n, then 

S(sm)(Xjv) • S(tn)(Xjv) = sx{Xn), 

A 

where the sum is taken over all partitions A with length <m + n such that 

+ ^m+n-i+l = S + t, i = 1, . . . , m, 

Am > max(s, t), 

^m+l = • • • = A^ = t. 

Apply the minor summation formula in Theorem 7 to 

G = Mp{Xn), H = MR{Xn), 

and the skew-symmetric matrix A = {aij)ij^r with nonzero entries 



Cl'i,a+b- 



1 if < i < (a + 6)/2- 1 
-1 if (a + 6)/2+ 1 < i < a + 6. 



Then, by the same argument as in the proof of [11, Theorem 2.4], we obtain the first formula 
in item (1) of the Theorem. The other formulas are obtained by applying Theorem 7 to the 
above skew-symmetric matrix A and the matrices 
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{G,H) = {Mp{Xn+l),MQ{Xn+l)), {Mp{Xn),MQ{Xn)), (Mp(X,+i), Mji(X,+ i)). □ 



Now we are in the position to complete the proof of Theorem 3. 



Proof of Theorem 3. Suppose that a and b are even. Combining (5.3), (5.4) and (5.7), and 
using (5.2), we have 



SxiXn+l)Si^{Xn) 

(A,/u)E7^(a,6) 

/ iNbn+n— b { iUn-6)^+b/2 



A(X,)A(X„+i) ' ' ' ' A(X,)A(X„+i) 

X det ^M[o,n-b/2-l](-^n) '■ M[n+a/2-b/2,n+a/2-l]{Xn)^ 

X det ^M[o,n-6/2-l](^n) '■ -^[n+a/2-6/2+l,n+o/2] (^n)^ 

X det ^M[o,n-6/2](-'^n+l) '■ -^[n+a/2-6/2+l,n+a/2] (-'^n+l)^ 

X det ^M[o,n-6/2-l](-'^n+l) ^ M[„+o/2_6/2,n+o/2] (-'^n+l)^ 
(-1)'' 

(^n)s^^a^i^f'/2^ (-^n)s^^a^b/2^ )« ^ ^ a ^ b/a+i^ (-^n+l)- 



If a and b are odd, then it follows from (5.3), (5.4), (5.8), and (5.2) that 
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(A,/u)e7^(o,6) 

/ -i\bn+n-b ( i \26(n-6)+(n-6)^ + (6-l)/2 

_ \.~^) ^_-^'^bn+b{b-l)/2+n-h _ ^_-^Y-b\~^) 



A(X,)A(X„+i) ^ ' ^ ' A(X,)A(X„+i) 

X det ^M[o,n-(6-l)/2-l](-'^n) '■ -^[n+(a+l)/2-(6-l)/2,n+(a+l)/2-l] (-'^n)^ 

X det ^M[o,n-(6+l)/2-l](-'^n) '■ M[„+(o+i)/2_(b+i)/2,n+(a+l)/2-l] (-'^n)^ 

X det ^M[o,n-(b+l)/2](-'^n+l) ^ M[„+(o_i)/2_(6+i)/2+l,n+(a-l)/2] (-'^n+l)^ 

X det ^M[o,n-(&+l)/2](^n+l) '■ -^[n+(a+l)/2-(&+l)/2+l,n+(a+l)/2l(-^n+l)^ 

S ^^a+l^^(b-l)/2-^ (^n)-g^^o + l-^(b+l)/2'j (-^n)-^ ^ ^ -J (b+l)/2-j ^ ^ a+1 -J (b+l)/2-J 

□ 
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